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ABSTRACT 


Controllers are the key component of software-defined network (SDN) 
architecture. Given the diversity of open SDN controllers, the following 
question arises for the network administrators: How can we choose 
the appropriate SDN controller? Different characteristics of the controllers 
have greatly increased the complexity of the right decision. Multi-criteria 
decision-making methods (MCDMs) isa family of robust mathematical tools 
to address complex problems regarding multiple objectives. In this paper, 
we study the most important features of SDN controllers. To this end, 
we compare the well-known SDN controllers including NOX, POX, Beacon, 
Floodlight, Ryu, ODL, and ONOS. Leveraging a novel MCDM technique 
called the best-worst multi-criteria (BWM), we find the most appropriate 
SDN controller. We solve an optimization problem and evaluate its 
performance in terms of significant criteria such as throughput and latency. 
Initial evaluation revealed that the ONOS and ODL have the highest 


throughput, while the lowest throughputs belong to the NOX, POX, and Ryu. 
However, final evaluation concerning all criteria confirmed the robustness 
of the ONOS and the ODL compared to other controllers. 
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1. INTRODUCTION 

Traditionally, network protocols are run inside the network devices. This architecture suffers from several 
problems, such as manageability, flexibility, and extensibility. To tackle the problems, the software-defined 
networking (SDN) concept has been proposed. In the SDN, the control plane of network devices is separated 
from the forwarding plane. The control plan is implemented centrally in a controller. The data plane performs 
packet forwarding, based on rules installed on the SDN controller. The SDN controller knows the global 
topology and information of the network. Therefore, it provides flexible and efficient management. 
The controller is programmable through applications communicating with it via the northbound application 
programming interfaces (APIs) [1-3]. 

The controller is the most significant part of the SDN architecture. Currently, there are a variety 
of open-source SDN controllers available for the community. The first SDN controller was NOX [4] which 
initially developed by Nicira [5]. Later, the NOX became the basic platform for many subsequent SDN 
controllers such as POX [6], ONIX [7], and Beacon [8]. Also, Floodlight [9], Ryu [10], OpenDayLight 
(ODL) [11], and ONOS [12] are the next generations of SDN controllers with more advanced features. 
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Due to the lack of standardization for SDN Networks [2], several platforms have been introduced by 
both the academic and industry sectors. These platforms have different features such as northbound 
and southbound APIs, single or hierarchical SDN controller model [13, 14], and supporting legacy network 
devices [15]. Given the inherent diversity of SDN controllers, the following question arises for the network 
administrators: How to choose the right SDN controller platform? In this paper, we suggest a new approach to 
answer this question. 

Due to diverse and different features, the selection of an SDN controller is a multi-objective 
problem. The well-known multi-criteria decision-making method (MCDM) is a robust mathematical tool to 
address complex decision problems involving multiple objectives [16]. In many works of literature related to 
computer networks such as [14, 15], various MCDM methods have been introduced. The most popular 
MCDM methods are multi-attribute utility theory (MAUT) [17], analytic hierarchy process (AHP) [18], 
analytic network process (ANP) [19], VIKOR [20], and Technique for the order of prioritization by similarity 
to ideal solution (TOPSIS) [21, 22]. 

Recently, Rezaei et al. [23, 24] proposed a novel MCDM technique called the (BWM) decision-making 
method. Compared to previous MCDM algorithms, this technique remarkably reduces the number of required 
comparisons. In this paper, we use the BWM algorithm to select the best SDN controller subject to network 
constraints. We run the so-called method over most well-known SDN controllers, namely the NOX, 
the POX, the Beacon, the Floodlight, the Ryu, the ODL, and the ONOS. The major contributions of our 
research are as follows: 

— We present a comprehensive study of both quantitative and qualitative features of the SDN controllers. 

— We give a comprehensive comparison of the most well-known controllers (NOX, POX, Beacon, Floodlight, 
Ryu, ODL, ONOS) considering both quantitative and qualitative features. 

— We analyze the best and worst features and then rank other criteria to select the best controller. 

— We use the BWM which is a novel MCDM technique to select the best SDN controller. 

— The proposed optimization approach leads to choosing the best SDN controller concerning user requirements. 

To the best of our knowledge, no research has so far deployed multi-objective optimization based on 
the BWM. Our proposed MCDM-based framework succeeds to find the best selection of SDN controllers 
considering both quantitative and qualitative criteria together. The organization of this paper is as follows: 
Section 2 describes the fundamentals and concepts of the problem and significant studies related to 
multi-objective SDN controller selection; section 3 presents the proposed scheme consisting of the proposed 
decision-making method in detail; in section 4, we present the results to rank the candidate SDN controllers; 
Finally, section 5 concludes the study and sketches future research directions. 


2. BACKGROUND AND LITERATURE REVIEW 

Simply speaking, SDN is a paradigm shift for traffic management by providing programmable 
control. Figure | shows the SDN architecture. There are three different layers: 

— Application layer: This layer provides service abstractions. It can be used by applications and services. 

— Control layer: This layer consists of a logically centralized controller which is the key component of the SDN 
architecture. The controller takes requests from the application layer and manages the forwarding planes via 
southbound API standard protocols such as the OpenFlow, and the OpFlex [25]. 

— Infrastructure layer: This layer consists of simple switches. When a switch receives a new flow, it sends 
a request to the controller. Then, the controller uses its global knowledge to find a path for the flow 
and sets the required rules down to the switches. 

The performance evaluation of a typical SDN controller includes scalability, reliability, efficiency, 
and robustness [26]. Our literature survey shows that almost all researchers use the CBench tool [27] 
to evaluate the performance of different SDN controllers. The CBench calculates the latency, the throughput, 
and the threading of the SDN controller. Zhu et al. [28] evaluated the performance of the NOX, the Beacon, 
and the Maestro [29] comparatively. In this paper, a version of NOX, namely NOX-MT, is developed to gain 
better performance. Tootoonchian et al. [30] performed the same study, but this time for the Floodlight, 
the Beacon, the NOX-MT, and the Maestro. In this study, the controllers are evaluated concerning 
the proposed architecture. Their results show that the performance of almost all SDN controllers varies based 
on their features such as switch partitioning, packet batching, and multi-threading. 

Shalimov et al. [31], presented a new tool, named HCprobe. The HCprobe is capable to compare 
seven different SDN controllers, namely the NOX, POX, Floodlight, Beacon, Ryu, MUL [32], and Maestro. 
The authors concluded that the proposed SDN controllers have several security vulnerabilities. They also 
reported that the Beacon is the most effective controller based on test results. Suh et al. [33] introduced 
almost the same tool for performance analysis of SDN controllers with more systematic measurements. 
In [34], latency and throughput for ODLare calculated under different scenarios. Their results show that 1) 
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the throughput of ODL is notably affected by the cluster size and 2) the controller failover time is dependent 
on the number of switches and the failure detection threshold. 
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Figure 1. The SDN architecture 


Salman et al. [35] analyzed the performance in terms of the latency and throughput metrics. 
They performed the analysis over the MUL, Beacon, Maestro, ONOS, Ryu, OpenDayLight, Floodlight, NOX, 
IRIS [36], MUL, Libfluid-Msg, and the POX under a varying number of switches and threads. 
Their results show that the MUL and the Libfluid-Msg have the best throughput performance, while the Maestro 
has the best latency. In [37], the authors did the same evaluation for Ryu, Floodlight, ONOS, and ODL. 
From the performance evaluation, ONOS showed the best throughput, while in latency mode, Ryu was the best. 

In [38], authors calculate the performance of the POX and the Floodlight by analyzing network 
throughput and the round-trip time (RTT) using Mininet [39]. They evaluated different topologies 
(single, linear, tree, and user-defined) to connect the switches. Their results illustrate that the Floodlight 
outperforms the POX, for both evaluations. Nguyen-Ngoc et al. [40] evaluated the performance of the ONOS 
controller by developing several modules for the OFCProbe [41] which is itself a benchmark tool. In this 
paper, four parameters are evaluated including network topology discovery time, topology change detection 
time, reactive path provisioning time, and asynchronous message processing time. 

Khondoker et al. [42] provided a set of comparison attributes such as virtualization, open-source, 
age and interfaces. The comparison was done using the analytic hierarchy process (AHP) which is an 
MCDM-based method. According to their analysis, the Ryu controller was chosen as the most suitable one. 
However, the assumed scale is subjective and changing the scale might lead to a different conclusion. 

None of the above-mentioned researches provide a comprehensive optimization method in such 
a way that could result in finding the best SDN controller based on multiple candidate criteria. As mentioned 
earlier, in this article we are going to solve a multi-objective optimization problem. Some authors of this 
article have valuable experiences in solving this kind of problem with metaheuristic methods. Interested 
readers can refer to [43-45] for deeper study. 


3. PROPOSED SCHEME 

Given the diversity of SDN controllers, the following question arises for the network administrators: 
How can we choose a proper SDN controller? In this section, we are going to present a comprehensive 
and systematic method using advanced tools for decision making to answer this question. Advanced tools are 
used because choosing a proper SDN controller among several choices with many different characteristics 
can increase the complexity of the right decision. On the other hand, a large variety of different problems 
emerging in computer networks engineering projects can sufficiently be tackled using the MCDM methods 
and its related techniques to make the right decision among multiple options. As said before, there have been 
introduced many MCDM techniques. Without paying attention to their differences, all of them firstly requires 
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the definition of options and criteria, and then demand a measure (e.g. weights) for assessing the relative 
significance of the criteria [16]. 

In this paper, we use the best-worst multi-criteria decision-making method (BWM), one of the latest 
members of MCDM family methods. The BWM was proposed by Rezaei [24] in 2015. It can calculate 
the weight of each criterion concerning different criteria based on pair wise comparisons. BWM has less 
complexity compared to other methods. Figure 2 illustrates the procedure diagram of the proposed scheme to 
select the best SDN controller among multiple controllers based on the preference of the best criterion over 
all other criteria. In the following subsections, we are going to propose each step of this diagram in detail. 


3.1. Identification of options 
In this section, seven most well-known SDN controllers have been compared to each other. 

Below, we give the list of these controllers, their origin and characteristics: 

— NOx: It was the first SDN controller developed by Nicira to be run on Linux distributions. It supports 
the OpenFlow protocol and is written in C language [4]. 

— POX: It is the next generation of the NOX controller. It is a python-based controller for fast development. 
Writing software for the POX controller is fast and very easy which makes it suitable for research 
purposes, demonstrations or experimentation [6]. 

— Beacon: Beacon is a fast, cross-platform, modular and Java-based OpenFlow controller that supports both 
event-based and threaded operations [8]. 

— Floodlight: The Floodlight is an enterprise-class, Apache-licensed, Java-based open-source OpenFlow 
Controller. It is supported by the Big Switch Company [9]. 

— Ryu: The Ryu is a python-based and open-source SDN controller providing an easy environment for 
developers to create new network management and control applications. The Ryu supports fully 
OpenFlow versions (1.0 to 1.5) and Nicira Extensions. All of the Ryu’s code is freely available under 
the Apache 2.0 license [10]. 

— OpenDayLight (ODL): The ODL is a modular SDN controller for customizing and automating networks 
of any size and scale. It was designed for a commercial purpose that addresses a variety of use-cases in 
existing network environments. One of the main key features of the ODL is supporting clustering 
and distributed data store architecture [11]. 

— ONOS: A key feature of the ONOS is that it can run as a distributed SDN controller; allowing it to use 
the CPU and memory resources of multiple servers. Multiple ONOS controllers can be linked to form 
an ONOS cluster with each controller manages a set of switches. In addition, it provides fault tolerance 
when the server fails [12]. 


Establishm ent of Decision Context 


Definition Objectives & Criteria 
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Figure 2. The procedure diagram of the proposed scheme 


3.2. Definition objectives and criteria 
SDN controllers have different and almost diverse features. We divide the features into two 
categories, including the “quantitative” and the “qualitative” ones. The qualitative features are: 
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APIs: As shown in Figure 1, the SDN has three layers: application, control, and infrastructure. It can be 
seen that the control layer manages the forwarding devices with southbound APIs such as the OpenFlow 
protocol [46], and the OpFlex [25]. The controller is also programmable with northbound APIs such as 
REST [47]. 

The OpenFlow Version: we consider the OpenFlow as a separate criterion because OpenFlow is known as 
a key component of the SDN architecture and it has been updating regularly in recent years. 
While other controllers do not support all versions, the OpenFlow protocol has different versions. 
Documentation: Most of the controllers discussed in this paper are Open-source. An open-source program 
allows everyone to change it based on custom-made requirements. Unfortunately, the majority of 
the controllers suffer from rich documentation. 

Regular update: Unfortunately, some SDN controllers do not provide regular update, which causes some 
problems for the developers’ community. 

Programming language: Controllers have been written using different programming languages, such as C, 
C++, Java, and Python. We use the programming language to write the applications in the controllers. 
Simplicity, having a rich library for networking, and user-friendliness are the most significant features for 
a programming language. 

Legacy Network: Generally, it is difficult to completely replace all legacy network devices with SDN- 
enabled ones. We expect that several legacy network devices can be used in migration toward 
the SDN [48]. Thus, translating the SDN controller policy for legacy network devices, and 
vice-versa, is a subtle task [49]. The SDN controller can provide this mechanism by using a legacy 
network management protocol such as the NETCONF [50]. 

Distributed Model: There are two general models for the controller, including single and distributed 
(logically centralized) [13]. With a single SDN controller, all switches connect to the same controller. 
This model suffers from two major drawbacks: the scalability and the single point of failure [51]. 
The distributed model is known as a good solution in which the load of the network can be shared among 
multiple controllers. Figure 3 shows an example of a logically centralized controller. In this figure, 
the controllers share their local views to form a global network view. In Figure 3, the controllers need an 
interface to share their information. These interfaces are called east-west-bound APIs. 

Modularity: The modularity maintains the integration and functionality of applications. High modularity 
allows the SDN controller to perform tasks faster in a distributed manner. 

GUI: TheGUI is a form of user interface that allows users to interact with network devices through graphical 
icons and visual indicators. It is an important option for network administrators to monitor the network. 






Global Network View 


App | asa. | App App Pa App App PE App 


Figure 3. An example of the logical centralized controller 


Table 1 presents a feature-based comparison among SDN controllers based on the above-mentioned 


qualitative features. Now that the qualitative features of SDN controllers were discussed, we proceed to 
explain the quantitative features. The quantitative features of SDN controllers are as follows: 


Throughput and latency: Controller bench marker (Cbench) is a good testing tool for OpenFlow 
controllers by generating packet-in events for new flows. The Cbench emulates a group of switches, 
which are connecting to a controller. It also sends packet-in messages and waits for flow-mods to get 
pushed down [27]. The Cbench is run in two modes: throughput and latency. In the throughput mode, 
it computes the maximum number of packets that could be handled by the controller. In the latency mode, 
it sends a packet-in message and waits for flow-mods to get pushed down. In this manner, the Cbench 
computes the total execution time. 
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We run our test on a system with core 17 CPU, 3.2GHzx8 and 12GB RAM. Also, all experiments 


were repeated 10 times for each size of the network. Table 2 shows the parameters of the Cbench command 


which we have used. The Cbench command is:./cbench-c localhost -p 6633 -m 10000 -1 10 -s 16 -M 1000 -t. 


Table 1. A feature-based comparison among SDN controllers 


NOX POX Beacon Ryu Floodlight ODL ONOS 
First Release 2008 2011 2010 2012 2012 2013 2014 
License GPL 3.0 Apache 2.0 GPL 2.0 Apache 2.0 Apache 2.0 EPL 1.0 Apache 2.0 
Multi-Thread No No Yes Yes Yes Yes Yes 
Platform Linux Linux Linux Linux Linux Linux Linux 
MAC MAC MAC MAC MAC MAC 
windows windows windows windows windows windows 
APIs OVSDB OVSDBOF OVSDB OVSDB, OF,  OVSDBOF OVSDB,OFREST,YA = OVSDB, 
OFREST REST OF REST REST NG NETCONFPCEP, OF 
REST NETCONFO BGPSNMP REST, 
FCONFIG NETCONF 
OF Version 1 1 1 1.0 to 1.5 1.0 to 1.3 1.0 to 1.3 1.0 to 1.3 
Documentation poor poor poor Medium Very Good Very good Medium 
Language C++ python Java Python Java Java Java 
Legacy No No No No No Yes Yes 
Network 
Category Single Single Single Single Single Distributed Distributed 
GUI Poor Poor Poor Medium Medium Very Good Very Good 
Regular Poor Poor Poor Medium Medium Very Good Very Good 
Updating 
Modularity Poor Poor Medium Medium Medium Very Good Very Good 


Table 2.parameters of the Cbench command 
controller hostname or IP address 


-C 


-p controller port for switches to connect to 

-l internal iterations per test 

-m duration of an internal iteration (in msec) 

-M number of MACs/hosts to emulate per switch 
-S number of switches to emulate 


-t throughput (vs. latency mode) 


Figure 4 shows the throughput of the Cbench program for different scales of the network. As stated 
before, the throughput is defined as the number of responses per second from the controller. As is evident in 
the figure, the ONOS has the highest throughput, while the lowest throughputs belong to the NOX, POX, 
and Ryu. Similarly, Figure 5 shows the latency of different controllers. As is evident in the figure, the NOX, 
POX, and RYU have lower latency compared to other controllers. 
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Figure 4.Thethroughput of the Cbench program for different scales of the network 
(thousand responses per second) 
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Figure 5.The latency of different controllers (ms) 


3.3. Comparison settings 
In this section, first, we explain comparison settings and then we consider the above-mentioned 

features as the most important criteria. To compare SDN controllers, we assign a level in a range [1, 4] 
to each controller for each qualitative criterion as follow: 
Level 4: Very Good 
Level 3: Good 
Level 2: Medium 
Level 1: Poor 

In the MCDM terminology, a matrix that shows the values of criteria is called the “decision 
matrix”. As is evident in the decision matrix of Figure 6, we augment both quantitative and qualitative 
criteria altogether. The level of each controller for each criterion is obtained from Figure 6. 
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Figure 6. A decision matrix containing the qualitative and the quantitative data 


MCDM methods involve several steps among them normalization is an important one. Since 
the normalization concept is used to eliminate the units of each criterion, all the criteria are dimensionless [52]. 
To normalize the decision matrix, we should notice that there are two types of criteria in the decision matrix, 
including quantitative and qualitative. We use different methods for the normalization of these two types. 
For normalization of the qualitative criterion x, in the decision matrix we use the following relation: 





Xi j ; 
Pi = = J ‘ Vij E {1,2,7} 


S (1) 


Since the magnitudes of changes in quantitative criteria (throughput and latency) are exponential, we use 
the logarithmic scale for normalization. To normalize the throughput criterion x; X;j in the decision matrix we 


use the following relation: 
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log, x; ; 
Pi, j Se, Vj € {1,2,...,n} 


2 
> log, Xt, j (2) 
k=l 


Similarly, to normalize the latency, we use 


log, x; ; 
py =l-— Ei yje{l2. n} 


D logs xij e 
k=l 


In (3), note that because lower values of latency are more desirable we have used a negative sign. Finally, 
the normalized decision matrix of qualitative criteria is shown in Figure 7. The normalized decision matrix 
of throughput and latency are shown in Figures 8 and 9, respectively. 


APIs OF Doc. Language Legacy Category GUI Regular Modularity 
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Figure 7. Normalized decision matrix 
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Figure 8. The normalized decision matrix for Figure 9. The normalized decision matrix for 
throughput criterion latency criterion 


3.4. Best worst method (BWM) 

A major problem to select the best controller is the different importance of each criterion. 
For example, for a controller, a regular update is much more important than its language. So we need to 
determine the importance weight of each criterion. In MCDM terminology, the importance of each criterion 
is called the “weight of the criterion”. There are different ways to determine the weight of each criterion 
such as entropy method and (BWM). In this paper we use the BWM method which includes five steps as 
following: 


Step 1: Determine a set of decision criteria. In this step, a set of criteria {c,, c,,...,c,}1S considered. In our 
study, the criteria are APIs(c,), OpenFlow Version(c, ), Documentation(c,), Language(c, ), 
Legacy Net(c,), Category(c, ), GUI(c, ), Regular Update(c, ), Modularity(c, ), Throughput(c,, ), 
Latency(c,, ). 


Step 2: Determine the best and the worst criteria. The best criterion has the highest importance. In our study, 
we consider OpenFlow version, regular update, supporting distributed model as the best criteria, 
while language is the worst criterion. Note that the best and the worst criteria are chosen arbitrary 
and based on the network characteristics. 


Step 3: Determine Others-to-Worst vector, like step 3 as follows: 
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AAG ssa.) (4) 


2° wn 


where a, denotes the preference of the criterion c, over the worst criterion c, -In our study, the vector A „is 


as follows: 
A,, =(7,7,2,1,6,7,3,8,5,4,4) (5) 


Step 4: Determine the preference of the best criterion over all other criteria. The resulting Best-to-Others 
vector would be: 


Aaa aia (6) 


? Bn 


where a,, denotes the preference of the best criterion c, over criterion c, . We should notice that for each pair 


d,, anda, we havea, =a,,/a,,, whicha,,is the preference of “the best over the worst” criterion. 


wi ? 


The Best-to-Others vector in our study is considered as follows: 
A, = (1.143, 1.143, 4,8, 1.33, 1.143, 2.66, 1, 1.6, 2, 2)” (7) 


Step 5: Find the vector of optimal weights (w,, w,,,....w,). To find the optimal weights of the criteria, 
the maximum absolute differences! w, /w ee land | w,/w,—a,, |, for all j, must be minimized. 


In other words, we should solve a min-max optimization problem as follows: 


(8) 


tM =],  w,20 forall j (9) 


ej 
Bj |? jw 





min max , | 











W, 
w 


j w 


The optimization problem in (8) and (9) is equivalent to the following problem: 














min é (10) 
st.\—2-a,|<é, forall j (11) 
Wy 
—-a,|<é, forall j (12) 
Wy 
2 W, =]; W; >0 for all j (13) 
J 


For any value of € , multiplying the first set of the constraints in (12) by w,and the second set of constraints 


by w,,, the solution space of (12) is an intersection of 4n-5 linear constraints. Therefore given values large 
enough for &, it can be concluded that the solution space is non-empty. By solving the optimization problem 
in (10)-(13), we obtain the optimal weights (w,, w,,,....w,)andé . 

Figure 10 shows the optimal weights of criteria for our study. As is evident in the figure, the best 


criterion (regular update) attains the most important weight compared to other criteria. Inversely, the worst 
criterion (language) attains the least importance weight. 
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Figure 10. Optimal weights for our study 


4. RESULT: RANK THE SDN CONTROLLERS 

Now, we proceed to select the best controller based on the decision matrix and the weights 
of criteria calculated in (10)-(13). The weighted linear combination (WLC) aggregation method multiplies 
each raw of the decision matrix by the weights of criteria and then sums the results as follows: 


(Wis Wooe5W,)| oe te (14) 
Aor es Aug 

Finally, we rank the best SDN controller according to the results. Figure 11 shows this ranking. 
As is evident in the figure, the ODL and the ONOS are the best controllers based on our criteria, while 
the POX, the NOX, and the Beacon are less desirable. Another interesting point is that ranking does not 
change for different scales of the networks. It is worth mentioning that the selection of the criteria 
(the best and the worst criteria)is subjective in our study and changing these parameters may lead to 
a different conclusion. As a result, the parameters in the BWM should be considered carefully based on 
the specific characteristics of the network requirements. 
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Figure 11. Feature-based comparison of the candidates SDN controllers 


5. CONCLUSION 

In this article, we targeted one of the most important challenges of SDN networks, namely controller 
selection. After reviewing previous studies along with their objectives and limitations, we divided the features 
into two categories, namely quantitative and qualitative. The quantitative features are throughput and latency 
that are calculated by the CBench tool. On the other hand, the most important qualitative features are APIs, 
Programming language supported by the controller, supporting legacy network, modularity, GUI, regular 
update, and documentation of the controller. We considered the quantitative and the qualitative features as 
the criteria to compare the most well-known SDN controllers. We modelled the SDN selection problem using 
a mathematical technique from the family of MCDM techniques, called the (BWM).It was observed that 
the proposed method could establish a better trade-off in terms of the above-mentioned criteria. The results 
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show that the ONOS and the ODL are the best controllers, while the POX, the NOX, and the Beacon are 
located at the bottom of the list. The optimization framework presented in this research could help the network 
administrators to find the most appropriate SDN controller based on the given network characteristics. 

Future studies are suggested to exploit machine learning methods, especially reinforcement learning, 
to solve the problem. Addressing this estimation can undoubtedly lead to more accurate modelling. 
The machine learning algorithms are usually used when the answer to a problem (judgment label) with 
a dataset is known in advance. For this purpose, a classifier must be designed so that for the input features, 
it can determine the best class label (the type of SND controller). 
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